1. Field of the Invention
The present invention relates to computer graphics and more specifically to a three-dimensional still and animated object representation obtained from photos of real objects and their geometrical representations, and to a method and apparatus for representation and rendering, using Binary Volumetric Octree.
2. Description of the Related Art
In the immediate future, high-quality rendering of three-dimensional (3D) objects at interactive speed will receive the primary emphasis in modern graphic systems. The demand for high-quality rendering of 3D objects necessitates effective algorithms to be devised for compression of the objects and transmitting them via communications networks in such fields as electronic commerce, computer games, science, engineering, medicine. Use of traditional polygonal models of 3D objects during the last tens of years to simultaneously meet all these demands has failed to give the desired result. Polygonal models have two major shortcomings: large volume (e.g., realistic models require tens of million triangles) and difficulty of constructing.
To overcome these difficulties, several approaches to 3D graphics were suggested in recent years. The most advantageous of them seem to be methods based on using images of objects, and methods based on using points instead of triangles in 3D space.
Image-based methods represent the given object as a set of imagesxe2x80x94xe2x80x98photosxe2x80x99 of the objectxe2x80x94totally covering its visible surface, and taken from several different camera positions. Besides, each such image is accompanied with corresponding depth-map which is an array of distances from the pixels in the image plane to the object surface. An advantage of such a representation is that reference images can provide high quality of the object visualization regardless of its polygonal model complexity, and can be compressed by usual image compression techniques without sacrificing much quality. In addition, rendering time is proportional to the number of pixels in the reference and output images and not the object complexity.
Disadvantages are due to the fact that obtaining depth maps for real life objects (e.g., sculptures) is rather complicated operation, as well as to insufficiently developed techniques of handling such representations.
Point-based methods represent an object as a xe2x80x98cloud of pointsxe2x80x99 without imposing explicit local polygonal structure. In this method, a set of images with a depth defines a set of points (having corresponding colors) on the object surface by translating each pixel of each reference image by the corresponding depth value in the direction orthogonal to the image plane. Hence image-based representations are a particular case of point-based representations. In the following we shall concentrate on image-based representations as they are closer to our approach.
In literature, the two aforementioned trends are described in references [1] to [13] describing such 3D object representation and rendering methods, as Relief Textures Mapping [1], Layered Depth Images [2], Layered Depth Image Tree [3], Qsplat [4], Surfels [5] and some other that have been known in prior art. In the following discussion of the prior art approaches, references will be made to the following publications:
[1] Manuel M. Oliveira, Gary Bishop, David McAllister. Relief Textures Mapping, Proceedings of SIGGRAPH ""00;
[2] Jonathan Shade, Steven Gortler, Li-wei He, Richard Szeliski, Layered Depth Images, Proceedings of SIGGRAPH ""98;
[3] Chun-Fa Chang, Gary Bishop, Anselmo Lastra. LDI Tree: A Hierarchical Representation for Image-Based Rendering, Proceedings of SIGGRAPH ""99;
[4] Szymon Rusinkiewicz, Marc Levoy. QSplat: A Multiresolution Point Rendering System for Large Meshes, Proceedings of SIGGRAPH ""00;
[5] Hanspeter Pfister, Matthias Zwicker, Jeroen van Baar, Markus Gross. Surfels: Surface Elements as Rendering Primitives, Proceedings of SIGGRAPH ""00;
[6] Chamberlain et al., Fast Rendering of Complex Environments Using a Spatial Hierarchy, Proceedings of Graphics Interface ""96;
[7] Grossman and Dally, Point sample rendering, Proceedings of Eurographics Workshops on Rendering Techniques ""98;
[8] Lischinski and Rappoport, Image-Based Rendering for Non-Diffuse Synthetic Scenes, Proceedings of Eurographics Workshops on Rendering Techinques ""98;
[9] M. Levoy and T. Whitted, The Use of Points as Display Primitives. Technical Report TR 85-022, The University of North Carolina at Chapel Hill, Department of Computer Science, 1985;
[10] L. Westover, Footprint Evaluation for Volume Rendering, Proceedings of SIGGRAPH ""90;
[11] C. I. Connolly. Cumulative Generation of Octree Models from Range Data, Proceedings of Intl. Conf. Robotics, pp. 25-32, March 1984;
[12] G. H Tarbox and S. N. Gottschlich. IVIS: An Integrated Volumetric Inspection System, Proceedings of the 1994 Second CAD-Based Vision Workshop, pp. 220-227, February 1994;
[13] Curless, B., Levoy, M., A Volumetric Method for Building Complex Models from Range Images, Proceedings of SIGGRAPH ""96;
[14] C. Bregler, Video Based Animation Techniques for Human Motion, SIGGRAPH ""00 Course 39: Image-based Modeling and Rendering; and
[15] Paul F. Debevec, Camillo J. Taylor, Jitendra Malik, Modeling and Rendering Architecture from Photographs: A Hybrid Geometry-and Image-based Approach, Proceedings of SIGGRAPH ""96.
The common problem with image-based methods is occurrence of holes in the resulting image. Unlike polygonal models that are xe2x80x98continuousxe2x80x99 in the sense that the object surface is linearly interpolated into the interior of all the polygons (normally, triangles), image-based and point-based representations provide xe2x80x98discretexe2x80x99 approximations of the object. In case of image-based representations, the object surface is, in fact, approximated with small colored squares, i.e. shifted pixels of reference images. When viewing direction differs substantially from the normal direction to each of the reference image planes, projections of the approximating squares generally do not completely cover the projection of the object surface. Let as call such holes the holes of the first type. Another source of holes in the resulting image for image-based representations is the fact that some parts of the surface may be not visible in all of the reference images, but become visible for some viewpoints (holes of the second type). These holes are due to insufficient information contained in a particular image-based representation.
Relief texture method [1] suppresses holes of the first type by using an analog of linear interpolation, which may lead to distortions and artifacts, since interpolation is performed in the two-dimensional (2D) projection of the object rather than in 3D space. More importantly, holes of the second type can only be treated the same way under this approach. Since the method of [1] uses only 6 reference images, that is, projections of the object on the circumscribing cube faces, this imposes serious restrictions on this method application to complex shapes when there exist points invisible from all six cube faces. This approach was chosen to maximize rendering speed, namely by using fast prewarping (geometric transformation equivalent to the change of viewing direction under the orthographic projection), but it leads to quality degradation.
Layered depth images (LDI) [2] are data structure designed to avoid the problem with holes of the second type. LDI is an image whose pixels contain all the object points projecting to a fixed location in the reference image plane. Fast prewarping algorithm of [1] applies here as well. However, problems with holes of the first type remain. Splatting (first introduced in [10]) is used to solve the problem of holes of the first type. Splat is a small two-dimensional rectilinear or elliptical surface patch, endowed with a certain color distributionxe2x80x94e.g. Gaussian, centered at the center of the patch, or constant. Disadvantage of the LDI method is in its nonsymmetry since the representation is based on a projection in a certain fixed direction. This leads to difficulties with hole filling for viewing directions greatly different from said fixed direction.
LDI tree [3] is an octree with an LDI attached to each octree cell (node). The advantage of having a hierarchical model is that not every LDI in the octree should be rendered. Those cells that are farther away are rendered in less detail by using the filtered points that are stored in the LDIs higher in the hierarchy. This representation was devised in order to overcome the nonsymmetry of LDI by using many reference images. However, the storage amount becomes very large: LDI tree for 512-by-512 image (obtained from 36 reference images) occupies 30 Mbytes as reported in [3], and about half this amount was the tree structure itself. As reported in [3], rendering time for this object is also large: 2-3 seconds per frame on Silicon Graphics Onyx2 with 32250 MHz MIPS R10000 processors (without using parallelism).
Yet another representation combining image-based data into a tree structure is recently designed Surfels method [5]. It deals with a specific tree [8] that is a layered-depth cube (LDC) where instead of single LDI tree nodes contain three LDI""s corresponding to three orthogonal planes. Results reported in [5] were obtained for original model containing 81000 triangles. Frame rate of 11 frames per second (fps) for 256-by-256 output buffer was obtained on Pentium III 700 MHz processor. Surfels are reference image pixels shifted by a corresponding depth vector. Tree structure is used to speed up computations for choosing visible elements. Hole filling is achieved by nearest-neighbor or Gaussian filtering. Splatting is implemented in this structure. High quality of the resulting image is attained at the cost of data content and speed restrictions.
Recently introduced representation of Qsplat [4] should also be mentioned, although it is rather point-based than image-based method. This approach uses hierarchical point structure based on nested balls. Elliptical splats of proper size are used at the rendering stage. However somewhat complicated and time-consuming truncated culling was used in [4]. The data structure is also more complex, and requires more time to process.
The idea and various implementation methods for obtaining octree structured 3D model from range data such as sets of depth images were developed in [1]-[12]. [13] deals with a construction of polygonal model from original data using octree. All the above relates to still 3D image-based representations. Speaking of animated 3D objects, it should be noted that only very few image-based methods were suggested for this problem so far. In [14] an idea of facial image modification for almost constant 3D face geometry is developed. This is applicable only to a restricted class of animated objects and is not animation of an actual 3D object. In [15] architectural scenes are animated with the aid of view-dependent texture mapping which reconstructs architectural views from various viewpoints on the base of a few photos.
Therefore, it is clear that an image-based representation allowing compact storage, fast rendering with high output image quality, and suitable for animation purposes is needed.
It is an object of invention to provide a three-dimensional object representation based on depth images, requiring relatively small storage space and allowing for fast and high quality rendering, in which the drawbacks of the discussed prior arts are reduced or eliminated.
It is a particular object of the invention to provide a method and apparatus for 3D object representation and rendering, allowing for fast computing of coordinate transformation, automatic determination of an order of imposing elements of Binary Volumetric Octree (BVO), and formation of splats of accurately defined size.
It is still further object of the invention to provide a method for compact representation of an animated 3D object, allowing for fast and correct rendering.
Compact data stream representation is effected by separating geometric and color components of the BVO and using binary representation of the octree for compression of the geometric component, and 2D image compression algorithms for compression of the color component.
The above result is achieved in a method for representation and rendering of a three-dimensional (3D) object in accordance with the invention, comprising the steps of: converting original data of a three-dimensional object into Binary Volumetric Octree (BVO) format, where color is attributed to each BVO vertex corresponding to coordinates of points belonging to object surface, said color being the average of colors of object surface points whose coordinates are inside a cube representing a three-dimensional image element, voxel, corresponding to given BVO vertex; rendering the BVO by hierarchical traversal of the octree from root vertex to leaves, so that to obtain local 3D coordinates of centers of the voxels corresponding to BVO leaves, transforming the obtained 3D coordinates into 2D coordinates of the voxel center and information of size of the projected voxel image, and generating and displaying, for each BVO leaf, a corresponding splat covering area of the projected voxel image and using color information, the 3D object being visualized by a plurality of the displayed splats.
The original data of a 3D object is a set of depth images, or polygonal model data, or a set of colored points.
Furthermore, the hierarchical traversal of the octree is performed in order from BVO voxels farthermost from a viewer to closer BVO voxels.
To minimize computational complexity of the conversion process, before the traversal of the octree the method includes computing:
fT(i,"sgr"i)=T2ixe2x88x921"sgr"i
where T is 4xc3x974 matrix of coordinate type transformation,
"sgr"i is any 4-dimensional vector whose elements are 0 or 1,
i is an integer taking a value from 0 to the height value of the BVO analyzed, and using the obtained results in the hierarchical transformation of 3D coordinates into 2D coordinates, while computing Fi for each BVO vertex:       F    i    =                    ∑                  j          =          i                n            ⁢                        f          T                ⁢                  (                      i            ,                          σ              j                                )                      =                            f          T                ⁢                  (                      i            ,                          σ              i                                )                    =                        F                      i            +            1                          =                              F            i                    ⁢                      (                                          σ                i                            ,                              F                                  i                  +                  1                                                      )                              
which gives, after traversal of the entire path from the BVO root vertex to leaves, for every BVO leaf specified by local coordinates (x,y,z):
Tv=F1("sgr"1,F2("sgr"2 . . . Fn("sgr"n) . . . ))
where "sgr"i is determined by the expression using binary record of coordinates:   v  =            (                                    x                                                y                                                z                                                1                              )        =                  (                                                            (                                                      σ                    x                    n                                    ⁢                                      xe2x80x83                                    ⁢                  …                  ⁢                                      xe2x80x83                                    ⁢                                      σ                    x                    2                                    ⁢                                      σ                    x                    1                                                  )                                                                                        (                                                      σ                    y                    n                                    ⁢                                      xe2x80x83                                    ⁢                  …                  ⁢                                      xe2x80x83                                    ⁢                                      σ                    y                    2                                    ⁢                                      σ                    y                    1                                                  )                                                                                        (                                                      σ                    z                    n                                    ⁢                                      xe2x80x83                                    ⁢                  …                  ⁢                                      xe2x80x83                                    ⁢                                      σ                    z                    2                                    ⁢                                      σ                    z                    1                                                  )                                                                                        (                                  0                  ⁢                                      xe2x80x83                                    ⁢                  …                  ⁢                                      xe2x80x83                                    ⁢                  01                                )                                                    )            =                        ∑                      i            =            1                    n                ⁢                              2                          i              -              1                                ⁢                      σ            i                              
The above technical result is achieved by a three-dimensional (3D) object representation and rendering apparatus, comprising 3D object original data generation means for generating original data of 3D object; 3D object original data conversion means for converting the 3D object original data into Binary Volumetric Octree (BVO) format, said 3D object original data conversion means being connected to the 3D object original data generation means, where color is attributed to each BVO vertex corresponding to object point coordinates, said color being the average of colors of object points whose coordinates are inside a cube representing a three-dimensional image element, voxel, corresponding to said BVO vertex; and BVO rendering means connected to the 3D object original data conversion means.
The 3D object original data generation means is a three-dimensional real-life object scanner for outputting a plurality of 3D coordinates of points, or 3D object polygonal model generation means, or depth image generation means.
Furthermore, the above technical result is achieved in a method for representation of an animated three-dimensional (3D) object, comprising the steps of: converting original data of each 3D object in a sequence of 3D objects into Binary Volumetric Octree (BVO) format, where color is attributed to each BVO vertex corresponding to coordinates of points belonging to object surface, said color being the average of colors of the object surface points whose coordinates are inside a cube representing a three-dimensional image element, voxel, corresponding to said BVO vertex, so that to generate, for the sequence of 3D objects, a sequence of BVOs corresponding to the animated 3D object.
The method further comprises projecting each BVO in the sequence of BVOs onto faces of a cube corresponding to BVO root vertex, so that to obtain for each cube face a video stream corresponding to images of the animated object and generate an octree byte stream, each byte in the octree byte stream corresponding to an octree vertex, and each bit in byte indicating the presence or absence, respectively, of a subtree for said octree vertex, where six video streams and the octree byte stream form together representations of the animated 3D object, the six video streams being stored in compact form using MPEG video compression format, and the octree byte stream being stored using entropic compression.
Furthermore, the animated 3D object is visualized by projecting six reference image streams on BVO to determine color for each BVO vertex and project the obtained colored BVO for visualizing the animated 3D object.